Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CRDT vocab #69

Closed
wants to merge 3 commits into from
Closed

Add CRDT vocab #69

wants to merge 3 commits into from

Conversation

NoelDeMartin
Copy link
Contributor

@NoelDeMartin NoelDeMartin commented May 26, 2022

I'm about to release a new app I'm working on, and I've created a new vocab to track changes in resources. I'm calling it "CRDT" because I intended to implement a CRDT (Conflict-Free Replicated Datatype) data structure, using the Solid POD as storage. I'm not entirely sure if this is technically a CRDT or not, so I would be fine calling this "history tracking" or something else as well. For more background on this, we discussed it in the Solid forum in this thread: Request for Comments: CRDTish approach to Solid.

Initially, I intended to host this myself at https://vocab.noeldemartin.com/crdt or something, but talking with @timbl he mentioned that maybe this would make sense as a w3.org vocab. So here it is. I'm not entirely sure if this gets merged how the actual w3.org/ns/crdt url is configured and if I'll be able to modify the html or something. Or is that just auto-generated from this repository?

For an example of data using this vocab, you can look at the following files which represent changes made to a document over time (the parts within [[ ]] are just placeholders that will be replaced during testing, so assume those are real values):

@pietercolpaert
Copy link

Addition comment complementary to your proposal

If

  • Every resource is immutable
  • Deletions happen via lazy deletion (flagging a resource as deprecated/deleted/unscheduled/invalidated by/...)
  • Updates can be full resource updates instead of partial representations

We would be able to implement the event sourcing architectural pattern here using Linked Data Event Streams and your CRDT vocabulary. Is this of interest?

On URI dereferencing

Instead of w3.org, you could also use w3id.org, which you can make dereferenceable through a pull request here: https://github.com/perma-id/w3id.org - and this is also how we do it for the Linked Data Event Streams vocabulary: https://github.com/perma-id/w3id.org/tree/master/ldes

Copy link
Member

@csarven csarven left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • What specs need this vocab?
  • Is the vocab intended to be Solid-centric?
  • Are there other tools that needs this vocab?
  • Is this vocab considered stable?
  • ...

https://github.com/solid/specification/blob/main/CONTRIBUTING.md#vocabulary-management

Copy link
Contributor

@timbl timbl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable at quick glance -except the default thing noted.

solid-crdt.ttl Outdated Show resolved Hide resolved
@NoelDeMartin
Copy link
Contributor Author

Thanks everyone for the comments so far!

@pietercolpaert

If

* Every resource is immutable

* Deletions happen via lazy deletion (flagging a resource as deprecated/deleted/unscheduled/invalidated by/...)

* Updates can be full resource updates instead of partial representations

We would be able to implement the event sourcing architectural pattern here using Linked Data Event Streams and your CRDT vocabulary. Is this of interest?

Technically we could say the operations themselves are immutable, but the resource they make reference to isn't. I talked about it in the forum thread, but TLDR I want my apps to be as interoperable as possible, so the data itself still needs to have a standard format following a known vocabulary (schema.org for example).

The operations are auxiliary to resolve conflicts between edits made in different devices (the app is offline-first). It also allows me to have a history of changes, but that's something I still haven't paid much attention to.

Instead of w3.org, you could also use w3id.org, which you can make dereferenceable through a pull request here: https://github.com/perma-id/w3id.org - and this is also how we do it for the Linked Data Event Streams vocabulary: https://github.com/perma-id/w3id.org/tree/master/ldes

Thanks for pointing this out, I didn't know this existed. I guess what's cool about this is that I could host the vocab in my own website for example, right?

What I don't like is that developers trying to read the documentation would be redirected, and it'd be a bit confusing for them. If they're trying to learn about w3id.org/crdt#Operation for example, they would get redirected to vocab.noeldemartin.com/crdt#Operation and I think it's confusing that the canonical url redirects. I'd prefer it if the canonical url returns the appropiate response directly.

@csarven

What specs need this vocab?

None, currently. I'm the only one using it in a new app I'm developing. Is it necessary that vocabs are related to a spec? I don't think schema.org has a spec, I thought it'd suffice if the vocab is self-describing in how it should be used.

Is the vocab intended to be Solid-centric?

Not necessarily, I suppose. Any context using semantic data to track changes made to resources would be able to use it. When I say "resources" I mean rdfs:Resource.

Are there other tools that needs this vocab?

Not so far.

Is this vocab considered stable?

Given that what the vocab describes is so fundamental, I don't think it'll change too much. And if it does, it'll probably be to add more operations for example, but I don't think the ones I included will change. I may be mistaken though, given that I haven't even released the app yet. So I don't think I could call it stable.

The thing is that by the point I release the app, I want the canonical url of the vocab to exist. That's why I'm submitting this now.

https://github.com/solid/specification/blob/main/CONTRIBUTING.md#vocabulary-management

I've read the documents linked there, but I must confess I'm a bit lost with all the jargon. Reading the document about creating new namespaces, I didn't see anything specific I should do so I just created the vocab looking at this other PR #68. Although I see they reference a specification, is that the problem?

NoelDeMartin added a commit to NoelDeMartin/soukai-solid that referenced this pull request Jun 10, 2022
@NoelDeMartin
Copy link
Contributor Author

I just updated the vocab taking into account some of @timbl's comments.

I've also added a couple of new things to handle deleting resources: a Tombstone class that can be used to indicate that something that was there isn't anymore, and a :deletedAt property that can be used either on the Tombstone to indicate when the resource was removed, or in the Metadata to soft delete resources.

@NoelDeMartin NoelDeMartin requested a review from timbl June 14, 2022 18:16
solid-crdt.ttl Outdated Show resolved Hide resolved
solid-crdt.ttl Outdated Show resolved Hide resolved
solid-crdt.ttl Outdated Show resolved Hide resolved
solid-crdt.ttl Outdated Show resolved Hide resolved
Co-authored-by: Ted Thibodeau Jr <tthibodeau@openlinksw.com>
Copy link

@jaxoncreed jaxoncreed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

This allows having deletedAt on Metadata to support soft deletes
@NoelDeMartin
Copy link
Contributor Author

Hey, thanks everyone for all the feedback.

I am about to release my app and this doesn't seem to be moving forward, so I decided to publish the vocab under https://vocab.noeldemartin.com/crdt/. If you want to follow along, I'm working on it in this repository: https://github.com/noeldemartin/vocab.

I'm closing this issue for now, if there's any interest in the future to add something like this to the official ontologies we can discuss it.

@pietercolpaert
Copy link

Aw! I do think this is quite useful! Just trying to wrap my head around possible alternatives and what should be further maintained in Solid. Really like this line of work @NoelDeMartin!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants